Method and system for providing information using a supplementary device

ABSTRACT

A method and system for providing access to information via a supplementary device is provided. User access to primary information via a primary device is monitored. Key information related to the primary content is obtained by extracting and analyzing metadata sources for the primary information. Then, supplementary information related to the primary information is obtained based on the key information. The supplementary information is provided for user access via the supplementary device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 11/823,005, filed on Jun. 26, 2007, which claims priority fromU.S. Provisional Patent Application No. 60/904,044, filed Feb. 28, 2007,both of which are incorporated herein by reference in their entirety forall purposes.

FIELD OF THE INVENTION

The present invention relates to providing information to devices, andin particular to providing information to electronic devices in anetwork.

BACKGROUND OF THE INVENTION

The number of networked electronic devices such as consumer electronics(CE) devices in home networks is on the rise and so is the amount ofdata stored therein. Consumers store and access several types of content(e.g., movies, photos, broadcast television (TV), cable TV programs)on/via such devices in their home networks. Further, the amount ofinformation available on the Internet via the World Wide Web is on therise.

Given the large amount of data/information available on a home networkand on the Internet, it is very likely that consumers (users) can findextensive information on the Internet related to at least a portion ofthe data stored on their home devices. Moreover, many users are asinterested in accessing information on the Internet as they are inaccessing data on their home networks. As such, there have been attemptsat providing information of interest to users.

Some existing approaches do enable display of commercial information ona TV screen, next to or in the place of, an ongoing TV program. However,this leads to disruption in viewing the TV program. There is, therefore,a need for providing information of interest to a user withoutdisrupting content being accessed by the user.

SUMMARY OF THE INVENTION

The present invention provides a method and system for providing accessto information via a supplementary device. In one embodiment thisinvolves monitoring user access to primary information via a primarydevice and extracting and analyzing data from metadata sources for theprimary information to obtain key information, then obtainingsupplementary information related to the primary information based onthe key information, and providing the supplementary information via asupplementary device for user access.

Obtaining supplementary information includes providing the user with oneor more key information related to the primary information, and uponuser selection of one or more key information, obtaining supplementaryinformation related to the selected key information. Obtaining thesupplementary information related to the selected key informationincludes searching external sources for result information related tothe selected key information. A user interface is generated whichincludes the result information for display. In one example, the userinterface is displayed on the supplementary device, while in anotherexample the user interface is displayed on the primary device under thecontrol of the supplementary device.

These and other features, aspects and advantages of the presentinvention will become understood with reference to the followingdescription, appended claims and accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a functional block diagram of an example networkimplementing aspects of the present invention.

FIG. 2 shows a functional block diagram for supplementary content accessusing a supplementary device, according to an embodiment of the presentinvention.

FIG. 3 shows a flowchart of a process content access using asupplementary device, according to an embodiment of the presentinvention.

FIG. 4 shows an example screenshot illustrating content provided to auser, according to the present invention.

FIG. 5 shows another example screenshot illustrating content provided toa user on a supplementary device, according to the present invention.

FIG. 6 shows an example implementation of a related-informationgenerating component.

FIG. 7 shows an example architecture for supplementary content accessusing multiple supplementary devices, according to an embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a method and system for providinginformation using a supplementary device, wherein the information isrelated to content being accessed via a primary device. In oneembodiment, the present invention provides a mechanism that enables auser to access primary information (e.g., content) via a primary device,and access supplementary information that is related to primaryinformation, via a supplementary device. The supplementary deviceinteracts with the primary device and provides supplementary informationrelated to the primary information being accessed via the primarydevice.

In one example wherein the primary device comprises a TV for viewing TVprograms, the present invention provides access to information relatedto a TV program on a supplementary device such as a portable device(e.g., PDA, cell phone, laptop), without disrupting the TV program onthe TV. Since the TV program is not disrupted, watching the TV programon the TV can continue while related information is provided on theportable device. A TV screen is typically not optimal for displayingtextual information such as web pages due to distance between a viewerand the TV screen. According to the present invention, in one example asupplementary device can be maintained close to a viewer for easierviewing of information (e.g., text on web pages) related to a TVprogram. Further, since the related information is displayed on thesupplementary device rather than the TV screen, viewing of the TVprogram can continue without disruption or change. Further, thesupplementary device provides more privacy for viewing the relatedinformation than if the related information was displayed on the TVscreen along with, or instead of, the TV program.

FIG. 1 shows a block diagram of an example system 10 such as a homelocal area network (LAN), implementing an embodiment of the presentinvention. The network 10 can be wired, wireless, or a combinationthereof. The network 10 includes a primary device comprising a computingmodule 12 and a supplementary device 14 such as a portable electronicdevice that can communicate with the computing module 12. The network 10can further include other devices such as a personal computer PC 17, aCE device 18, etc. One or more devices in the network 10 can implementthe Universal Plug and Play (UPnP) protocol for communicationthere-between, while other network communication protocols (e.g., Jini,HAVi, IEEE 1394, IEEE 802.11, Bluetooth) can also be used. The network10 is connected to an external network 50 such as the Internet, datasources, databases, content providers, etc. The computing module 12includes a TV 16 and a Set-Top Box (STB) 15 that receives a TV signal.The computing module 12 can also be the TV 16 itself or the STB 15 thatcan control the display on the TV 16. User activity in accessing contentis monitored. Data related to a program being watched on the TV 16 iscaptured by the computing module 12 using information sources includingmetadata such as closed-captions, electronic program guide (EPG), etc.Key information (e.g., keywords or key phrases) are extracted from suchdata and displayed to the user. Example techniques for capturingcontextual information from metadata sources and selecting keywords aredescribed further below.

The keywords are not only displayed on the TV 16 for the user but arealso displayed on the supplementary device 14. The supplementary device14 uses a protocol like HTTP, Bluetooth, etc., to obtain the keywordsfrom the computing module 12. The keywords can be used by a contextualdata extractor to form queries for finding additional information suchas web pages using search engine (e.g., Yahoo!), local and Internetmultimedia content such as songs and videos, or content such asdocuments from the PC. In one example, the keywords are displayed on thesupplementary device 14 and said web pages and content are provided onlyon the supplementary device 14. This can be suitable for configurationswhere the TV cannot access the Internet/home-network by itself or thecomputing module 12 cannot display graphics on the TV.

In another example, keywords are displayed on the TV 16 as well as thesupplementary device 14, and the user is provided the option to eitherview related web pages or content on the TV or the supplementary device14. When the user chooses to display the related content on the TV, thesupplementary device 14 downloads the content and displays it on the TV.When the user chooses to view the related content on the supplementarydevice 14, the content is either downloaded directly from the Internetor the computing module 12 downloads the content for the supplementarydevice 14 and passes it on to the supplementary device 14. The lattercase is useful in scenarios where the supplementary device 14 is notcapable of accessing the Internet.

Communication between the supplementary device 14 and the computingmodule 12 could be according to one of the following examples. In oneexample, the computing module 12 cannot contact the supplementary device14 but the supplementary device 14 can access the computing module 12.In that case, the computing module 12 places the information meant forthe supplementary device 14 in a certain storage location. Using alistening function, the supplementary device 14 polls the computingmodule 12 to check for such information, and then obtains theinformation from the storage location. In another example, thesupplementary device 14 can be accessed (such as where the supplementarydevice 14 includes a web server), wherein the computing module 12records the network location (e.g., an IP address in an IP network) ofand sends information to the supplementary device 14 at the networklocation. As those skilled in the art of networking will recognize,other processes for communication between the supplementary device 14and the computing module 12 are possible.

FIG. 2 shows an example architecture 20 for the computing module 12 andthe supplementary device 14 in the network 10, according to the presentinvention. The computing module comprises a user interface generatingmodule such as a web server 22, an application module 23 and acontextual data extractor 24. The supplementary device 14 includes a webbrowser 25, a media player 26 and an optional listener 27.

The computing module 12 has access to metadata related to activities ofthe user on the network 10, such as metadata related to content accessedby the user. For example, if the user is watching TV, the computingmodule 12 can access EPG information, closed-captions, etc., for the TVprogram being currently watched. The contextual data extractor 24extracts keywords from the metadata. In one example, keywords ofpotential interest to the user, based on, e.g., previous user activityin the network and/or the user profile, are extracted.

The web server 22 generates a user interface (UI) including thekeywords, and the UI is displayed by the browser 25 on a display in thesupplementary device 14. The keywords are available for display on theTV as well, as desired. Using the browser 25, the user can manuallyselect one or more keywords to obtain additional information through theInternet via the web server 22. The selected keywords are also used bythe application module 23 for obtaining additional related information.For example, the application module 23 implements one or more searchfunctions such as, e.g., an online video application that uses theextracted keywords to pose queries to YouTube to obtain contenttherefrom. In another example, the application module 23 can implement afunction that uses the keyword to obtain related news articles usingnews search engines on the Internet.

The default case is when the computing module 12 cannot contact thesupplementary device 14. In this case, the supplementary device 14 pollsor intermittently contacts the computing module 12 to determine if ithas additional information. When the optional listener 27 is present inthe supplementary device 14, the computing module 12 can inform thesupplementary device 14 directly of changes via a message. The listenerdirects the message to the browser 25 and the browser 25 updates thedisplayed web page with the information inside the message.

FIG. 2 further shows an example sequence of steps, illustrated by thearrows 1, 2 and 3, wherein in a first step, the supplementary device 14periodically sends a request for a UI to the web server 22 (polling),checking if the computing module 12 has new information. In response, ina second step if new information is available, the computing module 12responds with a positive acknowledgement, wherein in step 3 the urlgenerated by the web server 22 is displayed on the browser 25. In step 2when new information is present at the computing module 12, it is sentto the supplementary device 14 directly (the listener module 27 can becontacted).

The web server 22 generates the UI, such as an HTML UI, includinginformation obtained/searched by the application module 23, for use bythe supplementary device 14. The information obtained by the applicationmodule 23 is related to the TV program displayed on the TV. The browser25 displays the UI generated by the web server 22, for user viewing ofthe information obtained by the application module 23, and interactiontherewith. For example, where the information displayed on the browser25 includes a list of web-links obtained by the application module, theuser can use the browser 25 to navigate the web-links on the Internetthrough the web server 22, without disrupting the TV program displayedon the TV. The media player 26 (e.g., MP3 player, video player) in thesupplementary device 14 is used to play content accessed by the userthrough the browser 25.

The use of a web server and HTML for presentation is an example, andcommunication of information between the computing module 12 and thesupplementary device 14 can be performed in other ways such asBluetooth, etc. In another example, the web server 22 can dispense withcreating a UI, and simply provide the information obtained by theapplication module 23 to the supplementary device 14 to display. In thatcase the supplementary device 14 includes a UI generator module tolocally generate a UI for displaying information obtained by theapplication module 23.

FIG. 3 shows an example flowchart of a process 30 implemented by thearchitecture 20 in FIG. 2, wherein the computation device 12 (primarydevice) is implemented in the TV itself. The process 30 includes thefollowing steps in using the supplementary device along with watching aprogram on the TV:

-   -   Step 31: When a user wishes to find more information about a        program or a program excerpt being viewed on the TV screen, the        user activates an “Info” button on the TV or the supplementary        device.    -   Step 32: The computing module extracts keywords related to the        TV program using various metadata sources.    -   Step 33: The keywords are displayed on the supplementary device        and/or the TV.    -   Step 34: The user can choose between two options.

Step 34 a: Use the supplementary device as a remote control wherecontent related to the keywords is displayed on the TV but controlledusing the remote control. When a button/hyperlink of the web pagegenerated by the web server is activated via the browser in thesupplementary device, a request message (e.g., for downloading a videoor web page or new keywords) is sent to the computing module. Thecomputing module responds to the supplementary device and changes the TVdisplay to reflect the current state of the web page.

-   -   Step 34 b: Use the supplementary device as a control and viewing        device. When a button/hyperlink on the supplementary device is        activated, requesting keywords or information related to        keywords, the computing module sends a response back to the        supplementary device, and the graphics on the TV display is not        altered.    -   Step 35: The user selects keywords and a search application in        the application module.    -   Step 36: The selected search application obtains information        related to the keywords.    -   Step 37: The computing module generates a UI, including the        search results, for display, based on the viewing option        selected by the user in step 34.

The aforementioned hyperlinks are generated when a UI is generated bythe web server. The hyperlinks can be links to video or web pages on theInternet (when both the computing module and the supplementary deviceare connected to the Internet). Hyperlinks can also be requests for newkeywords from the computing module. When the supplementary device is notconnected to the Internet, all the hyperlinks are messages to componentdevices to download content or web pages or requests for new keywords,etc. The response from the computing module is a HTML page eithergenerated by the computing module or a page on the Internet, if thesupplementary device is not connected to the Internet.

As noted, the application module provides one or more searchapplications for obtaining information. In one example, the applicationmodule 23 in FIG. 2 maintains the following application information:

Application Name Query Interface Text Search Engine SearchText(q) VideoSearch Engine SearchVideo(q)

Then, the user uses the supplementary device 14 to choose an applicationfrom among the above applications, such as a video search engine. Then,when the user selects a keyword using the supplementary device 14, theapplication module 23 executes the method SearchVideo(q) where q is thequery (e.g., “Canada” for a TV news program on the North American FreeTrade Agreement) created by the contextual data extractor 24 based onthe selected keywords. In another example, q can be the selectedkeywords themselves. The SearchVideo method performs a search using thequery (e.g., on the Internet), and the application module 23 passes thesearch results to the web server 22 to construct a UI. The UI can bedisplayed on the supplementary device 14, and if desired, on the TV aswell.

In one example shown in FIG. 4, if the TV program is a news programabout natural disasters, and the related keywords generated by thecontextual data extractor 24 are “2004 Tsunami”, “Indian OceanEarthquake”, “Tsunami Warning”, and “Thailand”, then if the user selectsthe keywords “Tsunami Warning System” for searching and selects a textsearch engine application of the application module 23, then a UI 19created by the web server 22 includes text search results for thoseselections within the UI 19, as shown by example in FIG. 4.

In another example shown in FIG. 5, if the TV program is a news programabout natural disasters, and the related keywords generated by thecontextual data extractor 24 are “2004 Tsunami”, “Indian OceanEarthquake”, “Tsunami Warning System”, and “Thailand”, then if the userselects the keywords “Indian Ocean Earthquake” for searching and selectsa video search engine application of the application module 23, then aUI 19 created by the web server 22 includes thumbnails (and possiblytext) of found videos in the UI 19, as shown by example in FIG. 5, andcan be displayed on the supplementary device.

The application module 23 and the contextual data extractor 24 can beimplemented in various ways. FIG. 6 shows an example implementation 60for the application module 23 and the contextual data extractor 24, as asoftware component 62 that provides information to a client UI 64representing the web server 22 and the browser 25. The client module 64enables a user to request for more information from Internet datasources 66 about the TV program the user is watching. The component 62includes a Correlation Framework (CF) 106 and Data Extraction Plug-ins(DEP) 108. The CF 106 provides interfaces and functionalities for use bythe client module 64 to resolve user requests for various types of localand Internet information. The CF 106 includes a Query Execution Planner(QEP) 118, a Plan Library 124, a Correlation Plan Executor (CPE) 120,one or more RuleLets 126 and an Information Source Directory 127. Aplan-step in a plan specifies the RuleLet to be executed, and alsooptionally specifies the type of the required data and scope of theoutput data (e.g., whether the data required should be on the Internetor on the home devices). The RuleLets 126 are specialized processes thatexecute a specific task (e.g., obtaining keywords related to a TVprogram, obtaining Internet links containing the biography of anartist). The workspace 121 is used by the CF 106 to temporarily storethe results of executing each plan-step in a plan, for input tosubsequent steps as needed. The Information Source Directory 127maintains information about which data extraction plug-in should be usedto extract a particular type of data.

The QEP 118 provides interfaces between the client module 64 and thefunctions of the CF 106. The Plan Library 124 includes various plansdescribed in simple XML-like format. Each QEP 118 interface isassociated with a particular plan; when an interface is executed, theQEP 118 passes on the associated plan to the CPE 120 along with any datapassed on by the client/user. The CPE 120 executes the plan passed on bythe QEP 118 using the DEP 108. A plan specifies the steps to be executedin order to resolve a particular type of query. An example of the CF 106is described in related U.S. patent application Ser. No. 11/726,340,filed Mar. 21, 2007, entitled “A Framework for Correlating Content on aLocal Network with Information on an External Network,” incorporatedherein by reference. The DEP 108 provides functionalities to extractdata from different sources including the home devices/media repositoryand the Internet. The DEP 108 includes an Unstructured Data Extractor &Analyzer 102, a Semi-Structured Data Extractor & Analyzer 104, aStructured Data Extractor & Analyzer 110, a Broadcast Data Extractor &Analyzer 112 and a Local Content Gatherer 114.

The Unstructured Data Extractor & Analyzer 102 provides functionalitiesto extract data from general websites on the Internet via Internetsearch engines such as Google, Yahoo!, etc. The Unstructured DataExtractor & Analyzer 102 passes on the keyword chosen by the user/clientto a search engine (e.g., API provided by the ‘Yahoo!’ search engine) toobtain related web-links for the keyword. The Semi-Structured DataExtractor & Analyzer 104 provides functionalities to extract data fromInternet sources (such as zap2it) that provide semi-structured data(e.g., electronic program guide (EPG), internet movie database (IMDb)).The Structured Data Extractor & Analyzer 110 provides functionalities toextract data from Internet sources (e.g., compact disk database (CDDB))that provide structured music metadata. The Broadcast Data Extractor &Analyzer 112 provides functionalities to extract and analyze data frommetadata sources such as broadcast unstructured data sources 69 (e.g.,EPG, closed-caption data from broadcast TV programs). The Local ContentGatherer 114 provides functionalities to extract and aggregate metadatafor content (audio, video) available on local sources 67 (e.g., localcontent sources, local application states). In this example, the LocalContent Gatherer 114 in the DEP 108 and the CF 106 provide the functionof the contextual data gatherer 24. Further, the CF 106 and theremaining modules in the DEP 108, provide the function of theapplication module 23.

In one example, both the supplementary device 14 and the computingmodule 12 include hardware such as a microprocessor, memory andinfrastructure for communication. Software modules implement the stepsdescribed hereinabove, for execution by processors in the supplementarydevice 14 and the computing module 12, according to the presentinvention. Further, as shown by example architecture 70 in FIG. 7,multiple supplementary devices can be used in conjunction with a primarydevice, wherein each supplementary device can be used by a user toaccess information related to a program on the primary device,independent of the other supplementary devices.

As is known to those skilled in the art, the aforementioned examplearchitectures described above, according to the present invention, canbe implemented in many ways, such as program instructions for executionby a processor, as logic circuits, as an application specific integratedcircuit, as firmware, etc. The present invention has been described inconsiderable detail with reference to certain preferred versionsthereof; however, other versions are possible. Therefore, the spirit andscope of the appended claims should not be limited to the description ofthe preferred versions contained herein.

1. A method comprising: monitoring a media program on a media playbackdevice; extracting contextual program information related to the mediaprogram; analyzing the contextual program information to derive keyinformation; obtaining supplementary information related to the mediaprogram based on the key information; and sending the supplementaryinformation to a supplementary device for display on the supplementarydevice while the media program plays on the media playback device. 2.The method of claim 1, wherein the media playback device is atelevision.
 3. The method of claim 1, wherein the extracting includesretrieving data from a closed caption stream for the media program. 4.The method of claim 1, wherein the extracting includes retrieving datafrom an electronic program guide (EPG) object for the media program. 5.The method of claim 1, wherein the obtaining includes forming queriesfor the supplementary information based on the key information.
 6. Themethod of claim 5, wherein the queries are submitted to an Internetsearch engine.
 7. The method of claim 5, wherein the queries aresubmitted to a local content search mechanism.
 8. The method of claim 1,further comprising: modifying how the media program is played on themedia playback device based upon feedback received from thesupplementary device in response to the supplementary device displayingthe supplementary information.
 9. A method comprising: monitoring amedia program on a media playback device; extracting contextual programinformation related to the media program; analyzing the contextualprogram information to derive key information; and sending the keyinformation to a supplementary device to be used to retrievesupplementary information for display on the supplementary device whilethe media program plays on the media playback device.
 10. The method ofclaim 9 wherein the supplementary device comprises a handheld consumerelectronics (CE) device.
 11. The method of claim 9, wherein thesupplementary device displays the key information to a user to allow theuser to select specific key information, and wherein the specific keyinformation is used to retrieve the supplementary information fordisplay.
 12. A method comprising: receiving key information at asupplementary device from a media playback device, wherein the keyinformation is derived from contextual program information extracted bythe media playback device, wherein the contextual program information isrelated to a media program; retrieving supplementary information usingthe key information, wherein the supplementary information is related tothe media program; and displaying the supplementary information on thesupplementary device while the media program is played on the mediaplayback device.
 13. A media playback device comprising: a display; aninterface configured to receive a media program; a contextual dataextractor configured to extract contextual program information relatedto the media program; an application module configured to: analyze thecontextual program information to derive key information; obtainsupplementary information related to the media program based on the keyinformation; and send the supplementary information to a supplementarydevice for display on the supplementary device while the media programis displayed on the display of the media playback device.
 14. The mediaplayback device of claim 13, wherein the media playback device is atelevision.
 15. A media playback device comprising: a display; aninterface configured to receive a media program; a contextual dataextractor configured to extract contextual program information relatedto the media program; an application module configured to: analyze thecontextual program information to derive key information; and send thekey information to a supplementary device to be used to retrievesupplementary information for display on the supplementary device whilethe media program plays on the media playback device.
 16. The mediaplayback device of claim 15, wherein the supplementary device is ahandheld CE device.
 17. A supplementary device comprising: a display;means for monitoring a media program on a media playback device; meansfor extracting contextual program information related to the mediaprogram; means for analyzing the contextual program information toderive key information; means for obtaining supplementary informationrelated to the media program based on the key information; and means forsending the supplementary information to a supplementary device fordisplay on the supplementary device while the media program plays on themedia playback device.
 18. A media playback device comprising: means formonitoring a media program on a media playback device; means forextracting contextual program information related to the media program;means for analyzing the contextual program information to derive keyinformation; and means for sending the key information to asupplementary device to be used to retrieve supplementary informationfor display on the supplementary device while the media program plays onthe media playback device.
 19. A non-transitory computer readable mediumcontaining computer instructions stored therein for causing a computerprocessor to perform a method comprising: monitoring a media program ona media playback device; extracting contextual program informationrelated to the media program; analyzing the contextual programinformation to derive key information; obtaining supplementaryinformation related to the media program based on the key information;and sending the supplementary information to a supplementary device fordisplay on the supplementary device while the media program plays on themedia playback device.
 20. A non-transitory computer readable mediumcontaining computer instructions stored therein for causing a computerprocessor to perform a method comprising: monitoring a media program ona media playback device; extracting contextual program informationrelated to the media program; analyzing the contextual programinformation to derive key information; and sending the key informationto a supplementary device to be used to retrieve supplementaryinformation for display on the supplementary device while the mediaprogram plays on the media playback device.